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(54) Method and apparatus for generating moving characters 

(57) An avatar is generated by 



t) providing: 

a) a three-dimensional representation (2) of the 
structure of the character, the structure repre- 
sentation changing in real time to represent 
movement of the character, 

b) a three-dimensional representation of the 
surface (1) of the character which is mapped 
onto said structure representation, and 

c) a two-dimensional representation of fre- 
quently changing portions of said surface, eg 
the portions used to generate facial expres- 
sions and 

ii) combining said representations a) : b) and c) to 
generate a combined representation of the moving 
character. Because the frequently changing por- 
tions are represented only two-dimensionally, less 
processing power and ROM is needed to display 
the avatar. 
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Description 

The present invention relates to a method and ap- 
paratus for generating moving characters More partic- 
ularly the invention relates, in preferred embodiments, 
to a method and apparatus (particularly a programmed 
personal computer) for creating real-time-rendered vir- 
tual actors (avatars). 

Techniques exist for creating photo-realistic avatars 
in eg arcade games and for cinema production but these 
normally rely either 1) on 3D laser scanning which is 
very expensive and produces models that are very de- 
manding of processing power and memory and cannot 
be run on conventional personal computers (PC's), even 
those based on the Pentium chip, or 2) on Full Motion 
Video (FMV), which places filmed sequences in compu- 
ter-playable formats. Such techniques have the follow- 
ing major disadvantages: 

a) the appearance of the characters in FMV is flat 
and non-immersive 

b) the colours in FMV are washed out 

c) the development cost of both FMV and laser 
scanning is high 

d) the game play is tedious and linear, restricted to 
pre-filmed or pre-rendered scenes 

e) the ROM demands of both are excessive. 



Alternatively, developers intercut occasional pre- 
rendered scenes using non-photorealistic dummies, but 30 

a) being non-interactive, they inevitably interrupt 
gameplay 

b) they invariably lack realism 

c) very long development times are required. 35 

Furthermore this expedient does not avoid the in- 
flexibility and excessive ROM demands of FMV. 

Currently no games developers use real-time pho- 
to-realistic 3D avatars in games for the PC platform. It 40 
is thought that the reason for this is that current 3D scan- 
ning technology is inappropriate for this application. In 
particular, existing whole body scanners produce mod- 
els of sizes well in excess of 100,000 polygons, which 
cannot therefore be rendered in real time on a PC. A 45 
reduction by a factor of 100 in the number of polygons 
is required for such real-time rendering and the polygon 
reduction toots which can be used for this purpose do 
not give acceptable results. 

An object of the present invention is to ovecomeor so 
alleviate some or all of the above disadvantages. 

Accordingly in one aspect the invention provides a 
method of generating a moving character for a display, 
comprising the steps of i) providing: 

55 

a) a three-dimensional representation of the struc- 
ture of the character, the structure representation 
changing in real time to represent movement of the 



character, 

b) a three-dimensional representation of the sur- 
face of the character which is mapped onto said 
structure representation, and 
5 c) a two-dimensional representation of frequently 

changing portions of said surface, 

and 

ii) combining said representations a), b) and c) to gen- 
10 erate a combined representation ol the moving charac- 
ter. 

In another aspect the invention provides an appa- 
ratus for generating a moving character comprising: 

is j) memory means arranged to provide: 



a) a three-dimensional representation of the 
structure of the character, the structure repre- 
sentation changing in real time to represent 
movement of the character, 

b) a three-dimensional representation of the 
surface of the character which is mapped onto 
said structure representation, and 

c) a two-dimensional representation of fre- 
quently changing portions of said surface, 



and 

ii) processing means arranged to combine said rep- 
resentations a), b) and c) to generate a combined 
representation of the moving character. 

Preferred embodiments are defined in the depend- 
ent claims. 

In its preferred embodiments the present invention 
enables the creation of fully three-dimensional charac- 
ters that are capable of moving and behaving in a con- 
vincingly human fashion, interacting with their environ- 
ment on a data-driven level, supporting a variety of tra- 
ditional photographic and cinematic techniques, such as 
moving from long-shots to close-ups without break or 
transition, and which provide lip-synchronisation and 
sufficient facial expression to enable a wide range of 
conversation and mood, all within the existing con- 
straints of real-time rendering on PC's. 

In preferred embodiments in which the rapidly 
changing area of the character conveys speech and fa- 
cial expression, emulation of a range of phonemes and 
a range of eye and forehead movements is provided for. 
It has been found that by subdividing the face into upper 
and lower areas and substituting texture maps accord- 
ingly, an acceptably wide range of expressions and 
mouth movements can be conveyed in an economical 
fashion. 

In such embodiments the two-dimensional repre- 
sentations c) are for example of eye, mouth and prefer- 
ably also forehead regions of the face. These are pref- 
erably photographic in origin. Consequently even slight, 
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simple movements in the head convey a feeling of real- 
ism, as the viewer reads their own visual experience into 
the character in a way that cannot be done with any oth- 
er illustrative technique. 

Because the rapidly changing areas of the charac- s 
ter are represented two-dimensionally (and therefore do 
not involve recalculation of position as the character 
changes orientation, as would be required in the case 
of a wire frame model in which changes in expression 
are generated by movements of polygons beneath the 10 
surface), less processing power, memory and band- 
width (if the image is transmitted eg over the Internet) is 
required. 

In one embodiment the production path for the av- 
atars is as follows: 1S 

Photograph (in 35mm format) subjects head from 
front and side viewpoints, for geometry creation, placing 
a physical right-angled calibration target (a pre-marked 
set square) horizontally within the photographed image 

Photograph subject head-on running through a 20 
fixed list of phonemes and facial expressions. 

Photograph subject's body from two viewpoints for 
geometry creation 

Using 3d plotting software (eg Wireframe Xpress, 
previously used for CAD purposes), plot each individual 2s 
vertex and polygon by hand to describe the surface 
structure of the heads. 

Image re-touch all of the photographic texture- 
maps so that they synchronise in colour and lighting (eg 
using Aldus PhotoShop and various other graphics 30 
packages) 

* Sub-divide geometry and apply texture-maps to ap- 
propriate areas. 

Create back and front segments for all limb sec- 
tions, and retouch texture-maps 35 

Define inverse kinematics hierarchy and animate 
using captured motion ( eg using a combination of Flock 
of Birds, Puppeteer, and 3D Studio) 

Feed into a suitable 3D engine (eg MUMM-E) which 
runs under, for instance, Argonaut's BRender real-time 40 
3D engine. 

The result is believably human pseudo-video ren- 
dered on the fly at frame rates of 1 2 - 1 6 frames a second 
on an ordinary pentium PC. 

A preferred embodiment of the invention will now 45 
be described by way of example only with reference to 
Figures 1 to 3 of the accompanying drawings, wherein: 

Figure 1 shows the generation of a structure repre- 
sentation mapped onto a surface representation of so 
a human character shown A) in front view and B) in 
profile; 

Figure 2 shows the above character in front view 
with the frequently changing portions ( forehead, ss 
eyes and mouth) used for facial expression shown 
in different states, and 
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Figure 3 is a diagrammatic representation of the ap- 
paratus used for the capture of motion of the body 
of the above character. 

The detailed production procedure for heads and 
faces is as follows: 

Stage 1 : Photography 

I: Format: the camera negative format should be 
35mm, so that the application can consistently cal- 
culate the camera position. 

II: Lens: the focal length of the lens should be ideally 
around 100m; this allows for perspective flattening 
effects to be made use of. 

Ml: Lighting: Lighting should be diffuse and evenly 
balanced across both sides of the face, most pleas- 
ing results have been obtained with warmer imag- 
es. Photographing subjects under tungsten lighting 
using 1 00-400 ASA daylight balanced film has prov- 
en effective. 

IV: Triangulation and calibration: The subject is pho- 
tographed with a calibration target S as shown in 
Figure 1A and Figure IB. The calibration target 
works well when it runs under the subject's chin, 
with about 200mm (8 inches) vertical separation. In 
order for the face to be mirrored correctly the front 
of the target must be parallel to the face. The Z axis 
of the target should move back over the subjects 
left shoulder. 

The resulting photographs 1 are subsequently 
used to generate a three-dimensional representa- 
tion of the surface of the subject onto which is 
mapped a polygonal model 2 of the underlying 
structure of the subject which in use is manipulated 
in three dimensions by the computer running the fin- 
ished video game. This process will be explained in 
detail subsequently . 

V: Angles: best results are obtained with photo- 
graphs taken at right angles, with profile (0 degrees) 
and full-face (90 degrees). The profile shot should 
cover the entire head. 

VI: Mouth movements: the following movements 
and expressions should be included: 

1 . Eyes looking left 
2: Eyes looking right 
3: Sceptical look 
4: Laugh 
5. Wide eyes 
6: Raised eyebrows 
7: Mad/cross 
8: Sad look 
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9: Half blink (sleepy) 
10: Full blink 

11: "Em" - Phonetic M sound - 
12: "eff" - Phonetic F sound - 
13: "Oh" - Phonetic O sound - 
14: "Yew" - Phonetic U sound - 
15: "En* - Phonetic N sound - 
16: "Arh" - Phonetic R sound - 
17: "Ae" - Phonetic A sound - 

With these movements care must be taken to en- 
sure no extreme deformations of the jaw-line occur. 
Keeping the jaw lightly clenched whilst photographing 
the mouth movements reduces the risk of this. 

Typical expressions are illustrated in Figures 2A 
and 2B The forehead, eyes and mouth regions 3, 4 and 
5 are recorded two-dimensionally and their different 
states are stored for use by the finished game program. 
Such use can involve appropriate combinations of 
stoics (eg frown, eyes left, lips compressed) according 
to tho gnrne play eg in response to commands from the 
user ol the game 

Stigc 2 Face digitisation 

i Work with tho profile (0 degree) photo in the left 
rv*nj window nnd the full-face (90 degree) in the 

i r.»Qitiso the calibration target S first of all. A dis- 
crepancy Crin occur when minor mismatches be- 
tween calibration data in the target settings window 
nod puei vocations of the points in the images take 
puce The symptom is usually a misalignment be- 
tween tnc calibration grid and one point of the trian- 
qic oJten the triangle lies out of the calibration plane 
by rtrounc 15 to 25 degrees. The remedy is to se- 
■octrvcy lengthen or shorten one of the dimensions 
m ire t-trqet settings window, in quarter to half inch 
increments until the triangle lies in the plane of the 
grid 

i i The plotting of the 3d polygonal structure repre- 
icnvitoo 2 on the photographic representation 1 of 
t*^c sum^cc is iiijstrated in Figures 1 A and IB. On 
no t^icc :nc first line to plot is that of the profile. 
This s>r chj d extend from the curve of the throat, up 
t-Muugft tr»c* curve ol the chin, over the mouth, up 
utHJu-i the nu^c through to the bridge of the nose, 
rtod up the torehead to the hairline. This line 10 (Fig- 
ures lAand 1B) should be made up of 10-12 points, 
;md be <m open path. 

IV 'his first profile line can then be taken into Excel 
Id tc straightened in the x-axis. The process for 
wording with 3DP files in Excel is described later. 

V Nex* tho hnirtinc should be followed down to the 



base of the left ear, and along the jaw. Again, 10-12 
points are likely to be needed to keep this line 
smooth. This line should join the profile at the bot- 
tom of the chin. 

5 

VI : The first complex component is the eye. Plot this 
as a fiat plane, with a point in the centre - on the iris. 
This can be moved back and forth in the profile view 
to adjust for curvature. Plot open paths along the 
10 eyebrow, and down the eyelid. Connect these with 
faces. 

VII: The second complex component is the nose, 
particularly the underside. Make frequent use of the 
is perspective view tool to ensure that the line of the 
profile is not distorted. This is easy to do, and results 
in caricature-like geometry is in the final model. Be 
especially careful that the points under the nose in 
the two images relate. 

20 

VIII: Build the forehead with a horizontal ribbon run- 
ning across to the temple, just above the eyebrow. 
Use as few faces as possible, but ensure that the 
turn of the angle of the forehead is neither to swept 
2S or to sharp. 

IX: Follow the pit of the eye, from eyebrow to nose, 
use this line as a basis for the rest of the front of the 
face. 

30 

X: Extend a ribbon across the mouth, probably no 
more than two or three points. Use this to create a 
plane from nose to chin; do not over-model the 
mouth, as the animated texture maps will not fit the 
35 geometry if you do. 

XII: Join the rest of the face with triangles. When 
finished, you will have a completed left hand side to 
the face. 

40 

XIII: The next component is the side and back of 
head. The entire side and back can be extracted 
from the profile shot if careful The first step is to 
start a new project, and then reverse the position of 
45 the photographs, so that the full-face is in the left 
window, and the profile is in the right. 

XIV: Begin by plotting from the centre of the hairline 
around to the base of the jaw, as close as possible 
so to the ending point of the faces jawline. Finish this 
line with a point buried within the centre of the head. 

XV: Plot the outline of the ear. Use one, or if neces- 
sary two, points inside the ear to form faces in the 
55 same way as has been done with the eye. 

XVI: Build transverse ribbons through the hair, from 
the centre to the left edge of the head. 
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XVII: For the back of the head, Drop lines vertically 
down from the transverse crown-line towards the 
nape of the neck or the base of the hair. In order to 
establish which point is which when closing faces, 
use the select point tool (by clicking on a verter). s 

XVIII: The final component is the neck and shoul- 
ders. The point at which you stop building out from 
the head will be a decision made in accordance with 
the structure for the rest of the model. It may be ap- w 
propriate to limit this to the collar line, or extend the 
neck further into the throat and clavicle area. 

XIX: Start a new project, with the profile in the left 
window now, and the fullface in the right hand win- is 
dow. 

XX: The neck is essentially a cylinder, with sufficient 
modelling to fit into the base of the head and extend 
into the head. Three lines of 6 points running around so 
the neck should suffice. Once completed, close fac- 
es as before. 



Stage 3: Mirroring 

I: It is now necessary to mirror the face, head/hair, 
and neck. The face needs more attention than the 
other two components, in that we need to move the 
texture map as well as mirror the geometry. 

II: Mirroring involves a dos-command line utility 
called, unsurprisingly, mirror. The syntax for this is 
very simply: mirror filename.3dp newfilename.3dp. 
Perform this function on each component. 
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III: The mirroring process is not entirely stable. It is 
now necessary to take the new 3dp files into Excel 
or another spreadsheet (Excel is known to work) 
and recalculate certain vertex locations. The prob- 
lem with mirror is that it will reflect geometry around 40 
the project centre point, not along the weld line of 
the model. In addition, mirror does not move thepix- 
• el location of the vertex co-ordinates in the texture 
map. This results in the texture map being mirrored 
as well as the geometry, rather than it picking up 45 
new co-ordinates from the new geometry is pixel lo- 
cation. The effect is somewhat counter-intuitive! 

IV: .3dp files are ASCII based text files, with a struc- 
ture that can be defined as a space-delimited vari- so 
able. Excel will import this structure, with some ef- 
fort. 

V: Open the file in Excel. Specify the file type as a 
text file (with the extensions .txt, .csv and prn). Add ss 
*.3dp to the file extensions list, and select the face 
file. 



VI: Excel will launch one of its "wizard" utilities; on 
the first page of this, select edelimitedi as being the 
file type. On the second, turn off tab" and turn on 
"space 0 . On the third, select "text" as being the data 
type. 

VII: The file will open. Select all of columns A to J 
and change the column width to 20. Click on the 
"centre" icon. Often, an error message will occur 
saying "selection too large, continue without undo? 
' Answer yes. This file may now be worked on. 

VIII: Approximately one third of the way down the 
file is the points database, with XYZ coordinates for 
the vertices. The first three point values will relate 
to the calibration target - ignore them, points 4 to n 
will be the values for original half of the face; look 
at the left hand column, which represents the x val- 
ues. Note two things: first that the first half of the 
column has negative values, second that the first 
10-12 points havethe same value. These are the 
first set of points defined for the profile. Make note 
of this value. Move down the column until the points 
turn from negative to positive. Thes are the values 
for the mirored half-face x coordinates. These val- 
ues must be changed. 

IX: The formula for moving these is =SUM(value)- 
(2* x), where x is the value from your profile line. 
Change all the values for the second half of the col- 
umn using this. 

X: At the top of the file is the points database. This 
gives the values for the equivalent location in x-y 
pixel co-ordinates for the vertices in left and right 
pictures. There are four columns (apart from the 
numbering of the points) They read x-left hand y- 
left hand x-right hand y right hand. The column to 
■ be concerned with is the x-right hand. Again, locate 
the mirror portion by noting the discrepancies in val- 
ues, and identify the number given to the weld line. 
The same process as with the points database 
needs to be performed, but in this case the formula 
is SUM(y-value+y), where y is the vlue in pixels from 
the welding line. 

XI: Do the above for both the neck and hair/head, 
but do not mirror the pixel database - symmetrical 
hairstyles are easier corrected in 3d Studio. 

XII: The next step is to convert the 3dp files to 3d 
Studio files and assign texture vertices. Again a 
command line tool, called convert, is used. The syn- 
tax is convert filename. 3dp -r name of right-hand 
photograph.bmp matx.tga . 

XII: It may well be necessary to work on the tex- 
turemap files; the hair may need to be cloned out- 
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wards, the same for the neck in the neck texture, 
and also clean hair off the forehead by cloning 

XIII: Take files into 3ds, join and move vertices 
where necessary. 

Stage 4: Lip Synching 

I: Using the basic fullface image, move to Animator 
Pro. Load all of the subsequent facial movements 
as a flic file. 

II: Cut out the active area of the face, i.e. around the 
eyes and down to the mouth, taking care to include 
areas of skin that might also move. 

Ml: Add these cut-outs in the correct positions down 
onto the basic face of the subject, using the original 
fullface shot 

IV: Cut out the areas of the photograph (3, 4 or 5 - 
Figures 2A and 2B) that will move from frame to 
frame. For example cut out one of the moving eyes 
and paste it down onto the original area of the im- 
age. Also select some of the surrounding skin and 
using the C (clip) and L (lasso) tools. Pick up parts 
of the skin using the C (clip) and M (move) tools. 
Use the apostrophe mark to paste these into the 
final flic. 

V: The flic file should then contain all of the images 
from the original photoshoot of seventeen expres- 
sions, and if necessary any in-between frames that 
are required to smooth the facial movement. From 
this flic file, a render in 3d Studio can ensure that 
none of the fluid facial movements exceed the limits 
of the static facial geometry. For final export into 
Brender or Rendermorphics, individual frames can 
be generated. 

VI: Apply the final animated file (saved in eel for- 
mat) to the facial geometry in 3ds. Construct a hi- 
erarchy that links the face and the head/hair, and 
place its pivot point at the nape of the neck. This 
allows for convincing head movement, combined 
with photographically realistic facial expressions. 
Adjust imperfections accordingly. 

The body of the character can be animated and 
combined with the head by standard techniques. One 
way of capturing motion of the body is illustrated in Fig- 
ure 3 wherein magnets are mounted at the joints ol the 
subject's body 8 and their position detected by an elec- 
tromagnetic positon sensor 8 of standard type. 

The avatar derived from such a technique can be 
controlled and displayed by a computer game. 

The invention extends to any novel combination or 
sub-combination disclosed herein. In particular any pho- 



tographic representation of the rapidly changing areas 
of the character, whether or not two-dimensional, may 
be employed. 

5 

Claims 

1. A method of generating a moving character for a 
display, comprising the steps of 

10 

i) providing: 

a) a three-dimensional representation of 
the structure of the character, the structure 

is representation changing in real time to rep- 

resent movement of the character, 

b) a three-dimensional representation of 
the surface of the character which is 
mapped onto said structure representa- 

20 tion, and 

c) a two-dimensional representation of fre- 
quently changing portions of said surface, 

and 

25 ii) combining said representations a), b) and c) 

to generate a combined representation of the 
moving character. 

2. A method as claimed in claim 1 wherein said com- 
30 bined representation is generated in real time. 

3. A method as claimed in claim 1 or claim 2 wherein 
said combined representation is displayed interac- 
tively in response to commands from a user. 

35 

4. A method as claimed in any preceding claim where- 
in two or more such two-dimensional representa- 
tions c) of different frequently-changing changing 
portions of said surface are generated and are com- 

40 bined with said representations a) and b). 

5. A method as claimed in any preceding claim where- 
in the two-dimensional representation(s) c) is/are of 
portion(s) of facial expressions. 

4S 

6. A method as claimed in any preceding claim where- 
in said three-dimensional surface representation b) 
is derived from a photograph. 

so 7. A method as claimed in any preceding claim where- 
in said three-dimensional structure representation 
a) comprises a multiplicity of polygons. 

8. A method as claimed in any preceding claim where- 
ss in said combining step ii) is carried out by a program 
running in a PC and the character is displayed in 
real time on the display of the PC. 
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9- An apparatus for generating a moving character 
comprising: 

i) memory means arranged to provide: 

5 

a) a three-dimensional representation of 
the structure of the character, the structure 
representation changing in real time to rep- 
resent movement of the character, 

b) a three-dimensional representation of w 
the surface of the character which is 
mapped onto said structure representa- 
tion, and 

c) a two-dimensional representation of fre- 
quently changing portions of said surface, is 
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representation , 

b) a three-dimensional surface representation 
which is mapped onto said structure represen- 
tation, and 

c) a two-dimensional representation of fre- 
quently changing portions of the surface of the 
avatar. 

A method of generating a moving character sub- 
stantially as described hereinbabove with reference 
to Figures 1 to 3 of the accompanying drawings 



and 



ii) processing means arranged to combine said 
representations a), b) and c) to generate a com- 20 
bmed representation of the moving character 

10. An apparatus as claimed in claim 9 wherein said 
processing means is arranged to generate said 
combined representation in real time. 25 

11. An apparatus as claimed in claim 9 or claim 10 
**erc.n said processing means is providedwith us- 
*r -<»niroiir>d input means and is arranged to display 
*a.c combmod representation interactively in re- so 
soon so 10 commands from a user. 

12. An apparatus as claimed in any of claims 9 to 11 
when ts arranged to combine two or more such two- 
o.monsKX^I representations c) of different fre- 35 
quotly <hangmg changing portions of said surface 
with SH»d representations a) and b). 

13. An apparatus as claimed in any of claims 9 to 12 
whcre.n the two-dimensional representation(s) c) 40 
•vaic of portionts) of facial expressions. 

14 An apparatus as claimed in any of claims 9 to 13 
herein said three-dimensional surface represen- 
t.-mon bi is derived from a photograph. 4$ 

15. An apparatus as claimed in any preceding claim 
wf.cren^otfHee-dimensional structure represen- 
t -itKjn n ) cue r. t> u ± coordinates defining a multiplic- 

'*y of polygons SQ 

16. An iippiKcjtus as claimed in any of claims 9 to 15 
when c PC arranged to display the character in 
roal limo on its display. 



17. An avatar composing; 

a) nn animated three-dimensional structure 
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(54) Method and apparatus for generating moving characters 



(57) An avatar is generated by 
\) providing: 

a) a three-dimensional representation (2) of the 
structure of the character, the structure repre- 
sentation changing in real time to represent 
movement of the character, 

b) a three-dimensional representation of the 
surface (1) of the character which is mapped 
onto said structure representation, and 

c) a two-dimensibhal~representation of fre- 
quently changing portions of said surface, eg 

the portions used to generate facial expres- 
sions and 

ti) combining said representations a) : b) and c) to 
generate a combined representation of the moving 
character. Because the frequently changing por- 
tions are represented only two-dirnensionally, less 
processing power and ROM is needed to display 
the avatar. 
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